Test Suites for Evaluation in Natural Language Engineering 1 Introduction
نویسندگان
چکیده
The TSNLP project is funded by the CEC and started on 1 December 1993 with a duration of 23 months. We would like to thank our partners in the project: Dominique Estival, Kirsten Falkedal and Sabine Lehmann at ISSCO, Geneva, Eva Dauphin and Veronika Lux and Sylvie Regnier-Prost at Aerospatiale, Paris, and Judith Klein, Klaus Netter and Stephan Oepen at DFKI in Saarbr ucken, Germany. Lorna Balkan, Douglas Arnold and Frederik Fouvry Dept. of Language & Linguistics, University of Essex, Wivenhoe Park, Colchester, CO4 3SQ, UK
منابع مشابه
Test suites: some issues in their use and design
Evaluation has always been a subject of interest to the MT community. It has also been a source of grief, as witnessed by the damning ALPAC Report (see Pierce and Carroll, 1966). This report led to the virtual end of government funding for MT in the USA in the sixties since it concluded that there was no immediate prospect of MT producing useful translation of general scientific texts. However,...
متن کاملTowards systematic grammar profiling.Test suite technology 10 years after
An experiment with recent test suite and grammar (engineering) resources is outlined: a critical assessment of the EU-funded tsnlp (Test Suites for Natural Language Processing) package as a diagnostic and benchmarking facility for a distributed (multi-site) large-scale hpsg grammar engineering effort. This paper argues for a generalized, systematic, and fully automated testing and diagnosis fac...
متن کاملTest Suites for Quality Evaluation of NLP Products
Test suites are a useful evaluation tool for developers and users of NLP products. The paper gives an overview of the tsnlp design and methodology and describes how the tsnlp data and methodology can be used in practice to provide a reliable assessment method of the linguistic capabilities of NLP products.
متن کاملPerformance Analysis of MT Evaluation Measures and Test Suites
Many measures have been proposed for machine translation evaluation (MTE) while little research has been done on the performance of MTE methods. This paper is an effort for MTE performance analysis. A general frame is proposed for the description of the MTE measure and the test suite, including whether the automatic measure is consistent with human evaluation, whether different results from var...
متن کاملTarget Suites for Evaluating the Coverage of Text Generators
Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites. We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems. We find that they are of some interest, in helping inter-system comparisons and in providing ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 1995